Sensor Fusion to Determine Reliability of Autonomous Vehicle Operation

ABSTRACT

A method for an autonomous vehicle includes: receiving first object data from a first sensor module; receiving second object data from a second sensor module; comparing the first object data to the second object data; determining, based on comparing the first object data to the second object data, whether the first object data corresponds to the second object data; and in response to determining that the first object data does not correspond to the second object data, performing an action for the autonomous vehicle.

RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/194,722 filed Nov. 19, 2018, the entiredisclosure of which application is hereby incorporated herein byreference.

FIELD OF THE TECHNOLOGY

At least some embodiments disclosed herein relate to computing systemsfor autonomous vehicles in general, and more particularly, but notlimited to, the use of sensor fusion to determine the reliability ofoperation for sensors of an autonomous vehicle.

BACKGROUND

Autonomous vehicles typically include many sensors to assist incontrolling the autonomous vehicle. In some cases, the failure of asensor may cause an accident, collision, or near collision involving thevehicle.

Multiple sensor fusion is used in computing systems to manage imperfectinputs from an environment. Sensor fusion can use techniques such asartificial intelligence (AI), pattern recognition, digital signalprocessing, control theory, and statistical estimation. Data frommultiple micro-electromechanical systems (MEMS) can be fused to provideapplications with increased response and accuracy.

Advances in sensor technology and processing techniques, combined withimproved hardware, make real-time fusion of sensor data possible. Forexample, a system can combine and integrate MEMS accelerometers,gyroscopes, pressure sensors, magnetic sensors, and microphones intomulti-sensor combinations with on-board processing and wirelessconnectivity.

Depending on the application, fusion of data can take place in differentways, typically involving the level of complexity and the number ofelements being measured. Feature-level fusion involves features that areextracted from different sensor observations or measurements andcombined into a concatenated feature vector. Decision-level fusion takesinformation from each sensor after it has measured or evaluated a targetindividually.

Sensor fusion also can be centralized or decentralized depending onwhere the fusion of the data occurs. In centralized sensor fusion, dataare forwarded to a central location to be correlated and fused. Indecentralized sensor fusion, every sensor or platform has a degree ofautonomy in the decision making that fusion causes.

In an example of centralized fusion, clients forward all of the data toa central location, and an entity at the central location correlates andfuses the data. In an example of decentralized fusion, clients handleall fusing of the data. Combinations of centralized and decentralizedsystems also exist.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 illustrates an autonomous vehicle including sensor modules thatprovide data to a central processing device, according to oneembodiment.

FIG. 2 illustrates a sensor module including a sensor that provides rawdata to a processor, according to one embodiment.

FIG. 3 illustrates an autonomous vehicle including a central processingdevice that receives data from sensor modules, according to oneembodiment.

FIG. 4 illustrates a vehicle that collects data during operation,according to one embodiment.

FIG. 5 illustrates a vehicle including sensor modules that provide datato a central processing device and a control system, according to oneembodiment.

FIG. 6 illustrates a server that stores event data received from avehicle, according to one embodiment.

FIG. 7 illustrates an example of a vehicle including sensor modules andconfigured using an Artificial Neural Network (ANN) model, according toone embodiment.

FIG. 8 shows a method to determine whether object data from a firstsensor module corresponds to object data from a second sensor module,according to one embodiment.

FIG. 9 shows an autonomous vehicle controlled and/or configured inresponse to determining that object data from a first sensor module ofthe vehicle does not correspond to object data from a second sensormodule of the vehicle, according to one embodiment.

FIG. 10 is a block diagram of an autonomous vehicle including one ormore various components and/or subsystems, each of which can be updatedin various embodiments to configure the vehicle and/or perform otheractions associated with the vehicle.

FIG. 11 is a block diagram of a centralized autonomous vehicleoperations system, according to various embodiments.

DETAILED DESCRIPTION

Proper autonomous vehicle system operation is mission critical in manycases. For example, proper vehicle navigation is critical to avoidingcollisions with persons or other vehicles.

Practices regarding safe vehicle system operation are becoming moreregulated. For example, the vehicle industry is adopting standardizedpractices for designing and testing products. One example is ISO 26262,which provides an automotive-specific international standard thatfocuses on safety critical components. ISO 26262 is a derivative of IEC61508, the generic functional safety standard for electrical andelectronic (E/E) systems.

Increasing complexity in the automotive industry has resulted in furtherefforts to provide safety-compliant systems. For example, modernautomobiles use by-wire systems such as throttle-by-wire, in which adriver presses on an accelerator pedal and a sensor in the pedal sends asignal to an electronic control unit. The control unit analyzes severalfactors such as engine speed, vehicle speed, and pedal position. It thenrelays a command to a throttle body. Testing and validating systems likethrottle-by-wire present a technical problem.

Existing vehicles often use full-system redundancy in order to increasethe reliability of vehicle operation, and also to be compliant withautomotive safety and reliability standards as ISO26262. For example, inmany cases vehicles use dual or triple full-system redundancy in orderto increase system's reliability and safety. In one example, existingvehicles use dual or triple redundancy in identical components. Forexample, two or three cameras are used to detect if any of the camerasis not working correctly; two or three radars are used to detect if anyof the radars is not working correctly; and two or three lidars are usedto detect if any of the lidars is not working correctly.

However, the foregoing full-system redundancy presents the technicalproblem that hardware needs and expense are significantly increased dueto the full-system redundancy. Also, the complexity of maintenance isincreased due to the need to maintain several redundant systems.

Various embodiments disclosed herein solve the above technical problemsby using smart sensors that execute local sensor data analysis in thesensors themselves and then report on detected objects. Additionally, acentral sensor health analysis component (e.g., processor) determinesthe health state of the smart sensors by comparing the detected objectsas between the sensors (e.g., compare a first sensor to a secondsensor).

Although sensors such as cameras, radar, and lidar will not alwaysdetect all objects at the same time, it is expected that there will bestatistical matches between the objects detected by the sensors (e.g.,when looking at a data sample size that is taken over numerous objectdetection events). For instance, during daytime operation, it isexpected that most of the time (e.g., greater than a thresholdpercentage of events), the camera, radar, and lidar will detect andreport on objects (e.g., cars and pedestrians) that are in the samefield of view. Further, for instance, during nighttime operation, radarand lidar are expected to reliably detect and report on detected objects(e.g., cars) in the same field of view.

Determining that there is a failure to maintain a statistical detectionrelationship (e.g., a statistical correlation within a predeterminedlimit) between the sensors (e.g., smart sensors) can indicate that amalfunction exists in one of the sensors, or in one of the sensor fusionprocessing components (e.g., the processor performing statisticalcorrelation of object data). In response to determining that such afailure exists, various actions can be taken (e.g., operation of thevehicle can be changed). In one example, a safety hazard signal istriggered (e.g., an alert is provided to a passenger, and/or acommunication is sent to a central server that monitors vehicleoperation for several vehicles).

In one example, the action taken is disabling of the autonomous mode ofoperation of the vehicle. In one example, a person is required to takeover control of the vehicle. In one example, the action is automaticallystopping the vehicle, then disabling the autonomous mode.

Various embodiments herein provide the advantage of cost saving byavoiding full-system redundancy. Instead of replicating an entirevehicle operational system, a lower cost health analysis solution isprovided for the sensors and sensor fusion portion of the system. Thissolution is provided without needing sensor and sensor fusionredundancy. For example, only that portion of the system other than thesensors and sensor fusion processing components needs to be redundant.

In one example, one advantage can be reduced cabling cost for a vehicle.For example, providing an object data result from a sensor moduleinstead of needing to transmit high-speed raw data signals out ofsensors (e.g., as required with redundant raw video data feeds) reducesthe cabling requirements for the vehicle. For example, only the detectedobject needs to be transmitted (and not the raw data used to detect theobject). This significantly lowers the bit-rate data bandwidth requiredby the vehicle's cabling (e.g., this avoids the need for high-speedcabling to connect smart sensor modules to a central processingcomponent). This saves additional cost at the vehicle system level.

In one example, another advantage provided is a smaller-sized centralprocessing component. For example, object data comparison requiressignificantly reduced processing power as compared to processing rawsensor data feeds in the central processing component.

In various embodiments, full-system redundancy is avoided based onstatistical correlations of similar components (e.g., redundancy incomponents that are expected to generate the same or similar results incertain conditions in a statistical sense). For example, three differentcomponents (e.g., camera, radar and lidar) are expected to havecorrelations in their identified, reported objects in certain types ofconditions, but not in other types of conditions (e.g., sensor modulesreport same or similar data on objects such as cars and pedestrians inthe daytime, but not in the nighttime and/or for other different typesof objects). For example, the expected statistical correlations may varyby object type (e.g., radar or light reflective object versusnon-reflective object). Object types (for use in providing result datafrom a sensor module) can be stored in a database in, for example, asensor module that processes raw sensor data (e.g., from a lidar).

Various embodiments disclosed herein relate to systems and methods foran autonomous vehicle. The vehicle can be, for example, any of severaltypes of autonomous vehicle (e.g., a car, truck, aircraft, drone,watercraft, etc.).

In one embodiment, a method for an autonomous vehicle includes receivingfirst object data from a first sensor module and receiving second objectdata from a second sensor module. The first object data and the secondobject data are compared. For example, the comparison can be performedby a processor (e.g., CPU, GPU, or FPGA). Based on comparing the firstobject data to the second object data, a determination is made whetherthe first object data corresponds to the second object data. Forexample, this determination can be based on comparing a statisticalcorrelation to a threshold (e.g., whether a correlation within one ormore predetermined limits). In response to determining that the firstobject data does not correspond to the second object data (e.g., due toa failure of sensor components or sensor fusion processing), an actionis performed for the autonomous vehicle. For example, the action can bea change in control of an operational characteristic of the vehicle suchas a change in navigation or engine control.

In some embodiments, the operation or configuration of the vehicle iscontrolled based on a determination of a sensor and/or sensor fusionfailure. For example, one or more systems of the vehicle can becontrolled by a host system. Data regarding operation of the vehicle canbe collected by a memory device. The collected data is generated by theone or more systems. After being collected, the data is stored in thememory device.

In one embodiment, the autonomous vehicle includes sensors that collectsensor data. A non-volatile memory device of the vehicle is used toreceive various data to be stored (e.g., raw sensor data and/or resultsfrom processing of raw sensor data, such as data regarding a type orposition of a detected object). One or more computing devices cancontrol the operation of the vehicle. At least one of these computingdevices is used to control collection, by the non-volatile memorydevice, of data generated by the vehicle during operation. This data caninclude the sensor data. The computing device is, for example, acontroller of the memory device, and the controller executesinstructions in firmware of the memory device. After collecting thedata, the memory device stores the data (e.g., using non-volatile memorysuch as 3DXP memory).

FIG. 1 illustrates an autonomous vehicle 610 including sensor modules612, 614 that provide data to a central processing device 618, accordingto one embodiment. Each sensor module 612, 614 provides object data tocentral processing device 618. One or more other sensors 616 providesensor data to central processing device 618. In one embodiment, sensors616 provide raw sensor data and/or object data.

Central processing device 618 compares the object data received fromeach of the sensor modules 612, 614. In one example, the comparison isbased on detection of a single object in the field of view of sensormodule 612 and sensor module 614. In one example, the comparison isbased on a correlation determined based on several events, each eventrelated to detection of a different object. Central processing device618 uses the comparison of the object data to determine whether theobject data from sensor module 612 corresponds to object data fromsensor module 614.

If the object data matches, then central processing device 618determines that system operation is reliable and that each sensor module612, 614 is functioning properly. If the object data does not match,central processing device 618 determines that a system failure exists.For example, the failure may exist in either or both of sensor module612 and 614.

In one example, object data from sensor modules 612, 614 is determinedto match if the data satisfies a statistical correlation. In oneexample, a correlation is determined and compared to a predeterminedlimit or threshold.

In response to determining that the object data from each sensor module612, 614 does not match, central processing device 618 sends a signal toa host processing device 620. The signal indicates that a failure hasbeen identified.

In one embodiment, host processing device 620 performs one or moreactions in response to receiving the signal indicating identification ofa failure. In various embodiments, host processing device 620 sendscontrol signals to one or more vehicle systems 622 that change theoperation of one or more aspects of the autonomous vehicle 610. In oneexample, navigation, braking control, and/or engine control are changed.In another example, a configuration for one or more vehicle systems 622is updated. In one example, firmware for one of the vehicle systems 622is updated. In one example, at least a portion of the firmware isdownloaded from a central server using wireless communications.

In one embodiment, the action performed by host processing device 620 isinitiating performance of one or more diagnostic tests. For example, adiagnostic test of sensor module 612 and/or 614 can be performed. Inanother example, alternatively and/or additionally, a diagnostic test ofcentral processing device 618 can be performed.

In one embodiment, sensors 616 provide data to central processing device618 that is used to determine a context of operation of autonomousvehicle 610. For example, sensor 616 may provide data indicating whetherthe vehicle is currently operating in daytime or nighttime. Thisprovides contextual data that central processing device 618 can use inmaking a comparison of object data from sensor modules 612, 614.

In one example, central processing device 618 is a system-on-chip (SOC),field programmable gate array (FPGA), CPU, or graphics processing unit(GPU). In one example, central processing device 618 stores run-timedata in a volatile memory device(s) (e.g., a DRAM device). In oneexample, central processing device 618 can include logic circuitryimplementing at least a portion of various embodiments described herein.

In one example, host processing device 620 is a system-on-chip (SOC),field programmable gate array (FPGA), CPU, or graphics processing unit(GPU). In one example, host processing device 620 stores data in avolatile memory device(s). For example, the data stored is received fromvehicle systems 622. The host processing device 620 can be, for example,a processing core of a processor, an execution unit, etc.

In one example, the central processing device 618 and/or host processingdevice 620 can include hardware such as one or more integrated circuitsand/or discrete components, a buffer memory, a cache memory, or acombination thereof. The host or central processing device can be amicrocontroller, special purpose logic circuitry (e.g., a fieldprogrammable gate array (FPGA), an application specific integratedcircuit (ASIC), etc.), or another suitable processor.

In one example, each sensor module 612, 614 is a camera. Each camerashares a common field of view. It is expected during normal operationthat the cameras will detect the same object most of the time. If theobject data fails to match when compared to a predetermined threshold,it is determined that there is a system failure. In one example, thecompared object data is expected to exceed a threshold percentage (e.g.,60%) if the system is operating properly. In one example, the thresholdpercentage is 99% for events that occur during daytime, and 70% forevents that occur during nighttime.

FIG. 2 illustrates a sensor module 612 including a sensor 624 thatprovides raw data to a processor 626, according to one embodiment. Inone example, sensor module 614 is the same as or similar to sensormodule 612. In another example, sensor 616 is similar to sensor module612. In one example, sensor 624 is a camera. In another example, sensor624 is a radar or lidar sensor. Processor 626 is, for example, amicroprocessor, a GPU, and/or an FPGA.

Processor 626 executes one or more processes to evaluate the raw datafrom sensor 624. Processor 626 provides object data as a result fromthis processing. The object data is transmitted to a central processingdevice. The object data includes, for example, position data and/orobject type data. Other metadata characterizing the detected object mayalso be included.

Processor 626 accesses memory 628 during operation. Memory 628 is, forexample, volatile memory and/or non-volatile memory.

Memory 628 includes an object database 630. Database 630 includes alibrary of object types. Processor 626 can retrieve a predeterminedobject type from the library when providing a result from evaluating rawdata from sensor 624.

In one embodiment, neural network 632 is stored in memory 628. Neuralnetwork 632 can be used to provide one or more inputs to processor 626when it is evaluating raw data from sensor 624. In one embodiment,neural network 632 receives inputs from sensors 616 that are used fordetermining a context of operation for autonomous vehicle 610 as anoutput from the neural network 632. The output is provided to processor626 for use in evaluating raw data from sensor 624.

FIG. 3 illustrates an autonomous vehicle 170 including centralprocessing device 618 to receive data from sensor modules 178, accordingto one embodiment. Sensor modules 178 are an example of sensor modules612, 614.

In one embodiment, central processing device 618 receives object datafrom sensor modules 178. Based on a comparison of the object data,central processing device 618 sends a signal to host system 172indicating that a system failure has been detected. Host system 172 isan example of host processing device 620. In response to receiving thesignal indicating the system failure, host system 172 sends a signal tovehicle system 176 over bus 174 that causes vehicle system 176 to changean operational aspect of autonomous vehicle 170. In another embodiment,host system 172 can change an operation and/or update a configuration ofvehicle system 185. In another embodiment, data for updating softwareand/or a configuration of the vehicle system 185, host system 172,and/or vehicle system 176 is downloaded from server 188 using wirelessinterface 182 and stored in memory device 180.

Data is generated by various systems of the vehicle 170 including, forexample, vehicle systems 176, 183, 185. For example, one or more ofsensor modules 178 are used by vehicle system 176 and 183. Each systemmay perform processing on sensor data and then transmit the resultingdata (e.g., object data) to another component of the vehicle and/or maypass through raw sensor data to another component.

In one example, data from vehicle system 176 is sent to a host system172 over a communications bus 174. The host system 172 controls variousoperations of the vehicle system 176 (e.g., an image processing andrecognition system). Host system 172 sends commands over bus 174 tovehicle system 176. For example, a command can be used to control engineoperation or to implement steering control.

Host system 172 also can control other systems such as vehicle systems183 and/or vehicle system 185. Host system 172 can receive repliesand/or data from these systems over one or more communication paths ofthe vehicle 170 (e.g., bus 174).

In one embodiment, memory device 180 monitors communications on bus 174.Memory device 180 can collect selected data from bus 174. In oneexample, memory device 180 listens for certain predetermined types ofdata for extraction from bus 174. The extracted data is stored by memorydevice 180.

In one embodiment, memory device 180 stores collected data in apredetermined physical location 186. Physical location 186 maycorrespond to a physical position or location of the vehicle 170. Inother cases, physical location 186 may correspond to a particularpredetermined component or a type of component (e.g., a particular typeof memory storage media).

In one embodiment, memory device 180 collects data stored in a volatilememory 184. In one example, volatile memory 184 is used by host system172. For example, host system 172 can collect data from various systemsof vehicle 170. This data can be stored in volatile memory 184. In oneembodiment, memory device 180 collects data from the sensor modules 178.

In one embodiment, after data has been collected by memory device 180,the data is stored in memory device 180. The stored data can betransmitted at a later time to a server 188 over a wireless interface182.

In one embodiment, server 188 analyzes the collected data received frommemory device 180. Based on this analysis, one or more configurations ofvehicle 170 can be implemented or updated. For example, server 188 cansend a command to memory device 180 via wireless interface 182 regardinga change in configuration to implement in vehicle 170. Memory device 180can control the implementation of the configuration change.

In another embodiment, based on analysis of the collected data, server188 sends a communication to host system 172. In one example, thiscommunication can be sent via a wireless interface (not shown). Thecommunication can be, for example, a command to update a configurationof vehicle 170. This update can be implemented under control of hostsystem 172. In one embodiment, the update is performed in response to adetermination of system failure by central processing device 618.

In one embodiment, the collected data is stored in a non-volatile memorydevice (e.g., memory device 180). After storing, the data is removedfrom the non-volatile memory device and, for example, analyzed by aserver (e.g., a server in the cloud) (e.g., server 188).

In one embodiment, analysis of stored data is used to identify unsafeaspects or design of the software that operates the vehicle (e.g.,software that executes on host system 172 to control navigation). Inresponse to this analysis, control and/or configuration actions can beperformed for the vehicle (e.g., via wireless interface 182). In somecases, the same or similar control and/or configuration can be performedfor other vehicles (e.g., vehicles of the same type and/or using thesame or similar software). In various embodiments, the vehicle can be,for example, any of several types of autonomous vehicle (e.g., a car,truck, aircraft, drone, watercraft, etc.).

In various embodiments, the collected data can be stored by any ofseveral non-volatile memory systems of the vehicle (e.g., at physicallocation 186). An example of a memory system is a storage system, suchas a solid-state drive (SSD). In some embodiments, the memory system isa hybrid memory/storage system. In general, a vehicle can utilize amemory system that includes one or more memory devices. The memorydevices can include media. The media can be non-volatile memory devices,such as, for example, negative-and (NAND).

In one embodiment, a cloud service (sometimes referred to as simply thecloud) is used to receive the collected data from the vehicle and thento analyze the data. For example, when the automatic emergency brakingsystem of a vehicle is activated, event data (e.g., the location) ofthat braking event is transmitted to, for example, a server or othercomputing device in the cloud service.

In one embodiment, event data collected by memory device 180 is receivedfrom a vehicle and analyzed. For example, pattern recognition can beused on the received event data. In one case, machine learning is usedto recognize patterns or regularities in data. In some cases, a patternrecognition system can be trained from labeled training data (e.g.,supervised learning). In other cases, when no labeled data is available,other algorithms can be used to identify previously unknown patterns(e.g., unsupervised learning).

In one embodiment, an event occurring on the vehicle is detected basedon a measurement of a brake pedal in a vehicle. For example, the eventmay be identified based on a foot pressure or extent of movement ascompared to a predetermined threshold. In another embodiment, an eventis identified based on a rate of deceleration of the vehicle. Forexample, if a rate of deceleration exceeds a predetermined threshold,then an event (e.g., a near collision) is identified. In anotherexample, the rate of deceleration is observed over a predetermined timeperiod (e.g., the rate is averaged over the time period). In response todetecting this event, system memory data is downloaded. The detection ofsuch events can trigger collection of data by memory device 180.

In one embodiment, in response to analyzing event data, at least oneaction is performed. For example, a communication can be sent to vehicle170 (e.g., from which the collected data is obtained) to configuresoftware of the vehicle and/or disable one or more functions of thevehicle or software.

In one embodiment, in response identifying unsafe software (e.g., asdetermined based on pattern recognition or other analysis of collecteddata, such as object data from sensor modules 178), a server can performone or more actions. For example, the server can send a communication tothe current vehicle. In one case, the communication can cause thecurrent vehicle to take corrective actions, such as terminating anautonomous navigation mode, braking, or changing course, or updatingfirmware of the vehicle (e.g., via an over-the-air firmware update).

In one embodiment, in response to receiving a communication from aserver that has analyzed data collected by memory device 180, vehicle170 can switch off its autonomous driving mode, use a backup system,and/or activate a braking system to stop the vehicle.

In another embodiment, the cloud service can send a communication to aserver or other computing device that monitors an operating status forother vehicles (e.g., a central monitoring service). The communicationcan, for example, identify that a software component(s) is unsafe.

In one embodiment, the determination whether a vehicle has experienced abraking event and/or been involved in an accident can be based on datafrom one or more sensors or sensor modules of the vehicle. For example,data from an accelerometer of the vehicle can indicate a rapiddeceleration of the vehicle (e.g., deceleration exceeding a threshold).In another case, data can indicate that an emergency system of thevehicle has been activated, such as for example, an airbag, an emergencybraking system, etc. In some embodiments, any one and/or a combinationof the foregoing events can be deemed to be an event for whichdownloaded data is transmitted to a server. Data associated with theforegoing events can be collected by memory device 180.

In one embodiment, one or more sensor modules (e.g., sensor modules 178)on the current vehicle are used to obtain data regarding braking eventsand/or objects in the environment of the vehicle as it travels. Objectdata from the sensor modules and/or data generated based on analysis ofother sensor data and/or other types of data can be, for example,transmitted to the cloud service wirelessly (e.g., using a 3G, 4G, or 5Gnetwork or other radio-based communication system) after being collectedautomatically by memory device 180.

In one embodiment, in response to identifying unsafe software determinedbased on collected data, one or more actions of a vehicle areconfigured. For example, an over-the-air firmware update can be sent tothe vehicle for updating firmware of a computing device of the vehicle(e.g., this update causes the vehicle to avoid identified unsafefunctions). In one example, the firmware updates a navigation system ofthe vehicle.

FIG. 4 illustrates a vehicle 200 that collects data during operation,according to one embodiment. Vehicle 200 is an example of autonomousvehicle 610.

For example, autonomous vehicle 200 detects various objects duringtravel. Vehicle 200 also detects certain events that trigger downloadingdata (e.g., from system memory) to a non-volatile memory device. In oneexample, vehicle 200 detects the event that a collision with an object(e.g., another vehicle) has occurred. In response to detecting thecollision, a computing system of vehicle 200 causes system memory datato be downloaded to a non-volatile memory device. This is done to permitanalysis of the downloaded data after the collision. In one example, anyof the foregoing data can be collected automatically by memory device180.

More particularly, in one embodiment, FIG. 4 illustrates a sensingsystem located on board vehicle 200 for collecting informationconcerning operation of the vehicle. Any of this collected data can be,for example, a portion of the data downloaded from system memory and/orfrom other volatile memory in response to detecting an event, orroutinely collected during normal operation of the vehicle.

For example, the sensing system can be used to collect object and eventdata for sending to a cloud service to be stored for analysis. Theobject data can also be sent to a central processing device to becompared to other object data, as described above.

The sensing system, in various embodiments, may generally include one ormore sensor modules 220, a processor 230, memory 240, and acommunication device (e.g., transmitter 250). Other data in addition toobject data can be collected for sending to the cloud service.

The sensing system, in various embodiments, may include one or moresensor modules 220 (sometimes referred to herein as simply sensors 220)configured to collect information regarding operational aspects ofautonomous vehicle 200, such as speed, vehicle speed, vehicleacceleration, braking force, braking deceleration, and the like.Representative sensors configured to collect information concerningoperational driving characteristics may include, without limitation,tachometers like vehicle speed sensors or wheel speed sensor, brakepressure sensors, fuel flow sensors, steering angle sensors, and thelike.

The sensing system, in various embodiments, may additionally oralternatively include one or more sensors 220 configured to collectinformation regarding the static environment in which autonomous vehicle200 is operated, such as the presence and content of physical objectssurrounding the vehicle. The physical objects include, for example,signage and traffic signals (e.g., stop signs, construction zones, speedlimit signs, stop lights), road lane dividers (e.g., solid and dashedlane lines), and the like. Representative sensors configured to collectsuch static operating environment information may include outward-facingcameras positioned and oriented such that their respective fields ofview can capture the respective information each is configured tocollect. For example, a camera configured to capture surrounding signagemay be configured towards the front of or on top of autonomous vehicle200 and oriented forward-facing (e.g., straight ahead or perhaps cantedsideways by up to about 45 degrees) so as to capture roadside andoverhead signage/traffic signals within its field of view as autonomousvehicle 200 travels forward.

As another example, cameras configured to capture road lane dividers maybe positioned on the side of or off a front/rear quarter of autonomousvehicle 200 and may be oriented somewhat downwards so as to capture roadlane dividers on both sides of vehicle autonomous 200. Additionalrepresentative sensors for collecting static operating environmentinformation may include receivers configured to receive wireless signalsfrom base stations or other transmitters communicating information thatmay ordinarily be found on signage or otherwise related to the staticoperating environment of autonomous vehicle 200. Likewise, globalpositioning system (GPS) or other location-related sensors may beutilized to collect information regarding the static environment inwhich vehicle 200 is operated, such as what street autonomous vehicle200 is driving on, whether that street is a traffic artery (e.g.,highway) or other type, and whether that location is in an urban orrural area.

The sensing system, in various embodiments, may additionally oralternatively include one or more sensors 220 configured to collectinformation regarding the dynamic environment in which autonomousvehicle 200 is operated, such as information concerning the presence ofmoving objects such as, for example, other nearby vehicles (e.g., alongwith such vehicle's location, direction of travel, rate of speed, andrate of acceleration/deceleration), as well as similar informationconcerning the presence of nearby pedestrians. In one embodiment, any ofthe foregoing information can be used during evaluation by a sensormodule 612, 614 to generate object data.

Representative sensors configured to collect such dynamic operatingenvironment information may include outward-facing cameras positionedand oriented such that their respective fields of view can capture therespective information each is configured to collect. For example,outward-facing cameras may be positioned about the perimeter ofautonomous vehicle 200 (e.g., on the front, rear, top, sides, and/orquarters) to capture imagery to which image processing techniques suchas vehicle recognition algorithms may be applied. Additionally oralternatively, one or more optical sensors (e.g., lidar, infrared),sonic sensors (e.g., sonar, ultrasonic), or similar detection sensorsmay be positioned about the vehicle for measuring dynamic operatingenvironment information such as distance, relative velocity, relativeacceleration, and similar characteristics of the motion of nearbypiloted or autonomous vehicles.

The sensing system, in various embodiments, may leverage as sensor(s)220 those sensors typically found in most autonomous vehicles such as,without limitation, those configured for measuring speed, RPMs, fuelconsumption rate, and other characteristics of the vehicle's operation,as well as those configured for detecting the presence of other vehiclesor obstacles proximate the vehicle. Sensors 220 may additionally oralternatively comprise aftermarket sensors installed on autonomousvehicle 200 for facilitating the collection of additional informationrelated to driving style.

The sensing system of vehicle 200, in various embodiments, may furthercomprise an onboard processor 230, onboard memory 240, and an onboardtransmitter 250. Generally speaking, in various embodiments, processor230 may be configured to execute instructions stored on memory 240 forprocessing information collected by sensor(s) 200 for subsequenttransmission offboard vehicle 200. Onboard processor 230, in variousembodiments, may additionally or alternatively be configured to executeinstructions stored on memory 240 for processing information from two ormore sensors 220 to produce further information concerning objectcharacteristics detected by autonomous vehicle 200. For example, in anembodiment, processor 230 may process operational characteristics, suchas braking deceleration, alongside dynamic environment characteristics,such as following distance, to avoid debris or an animal suddenlyappearing in the roadway. It should be recognized that this is merely anillustrative example, and that one of ordinary skill in the art willrecognize further ways sensor data may be processed by a processor toproduce further information concerning objects detected by autonomousvehicle 200.

Processor 230, in various embodiments, may be configured to pre-processinformation from sensor(s) 220 for subsequent offboard transmission viatransmitter 250. Pre-processing activities may include one or acombination of filtering, organizing, and packaging the information fromsensors 220 into formats and communications protocols for efficientwireless transmission. In such embodiments, the pre-processedinformation may then be transmitted offboard vehicle 200 by transmitter250 in real-time or at periodic intervals, where it may be received bynearby piloted or autonomous vehicles and/or a remote server. It shouldbe appreciated that transmitter 250 may utilize short-range wirelesssignals (e.g., Wi-Fi, BlueTooth) when configured to transmit thepre-processed information directly to nearby piloted or autonomousvehicles, and that transmitter 250 may utilize longer-range signals(e.g., cellular, satellite) when transmitting the pre-processedinformation directly to the remote server, according to variousembodiments later described.

In one embodiment, processor 230 can be a processor of a memory device(e.g., memory device 180) that operates independently of a host systemto automatically collect data from vehicle 200.

Like sensor(s) 220, in various embodiments, processor 230 and/or onboardtransmitter 250 may be integrally installed in vehicle 200 (e.g., carcomputer, connected vehicles), while in other embodiments, processor 230and/or transmitter 250 may be added as an aftermarket feature.

FIG. 5 illustrates a vehicle including sensor modules 737 that providedata to central processing device 618 and a control system 731,according to one embodiment. Sensor modules 737 are an example of sensormodules 612, 614.

In one embodiment, control system 731 is used to download data inresponse to detecting an event associated with a vehicle 711, accordingto one embodiment. In one example, the detected event is a systemfailure determined based on a signal received from central processingdevice 618. In one example, central processing device 618 receivesobject data from each of sensor modules 737.

In one example, the control system 731 is included in memory device 180of FIG. 3, and the downloaded data is collected by memory device 180.

Control system 731 also controls at least one operation (e.g. navigationand/or braking) of the vehicle. The control system 731 includes one ormore processors 733, non-volatile memory 735, and system memory 729,which includes volatile memory 727. System memory 729 may also include,for example, data of vehicle 711 stored in other volatile memory devices(not shown).

The volatile memory 727 may be, for example, a dynamic random accessmemory device. Volatile memory 727 is used to store data used bysoftware during operation of the vehicle such as sensor data (e.g.,object data) and data regarding the operational state of the controlsystem 731. The object data is collected, for example, by sensor modules737.

Non-volatile memory 735 is used to store data from the volatile memory727 after an event is detected. Examples of non-volatile memory 735include 3D XPoint memory and NAND flash memory, as well as removablememory devices such as solid state disks.

The vehicle 711 uses sensor modules 737 and control system 731 to detectvarious events, as mentioned above. These events can include, forexample, the collision of vehicle 711 with an object. Other events maybe detection of a system activity such as emergency braking systemactivation or sensor data exceeding a threshold. When an event isdetected, data stored in volatile memory 727 is downloaded tonon-volatile memory 735. Examples of the stored data includes sensordata such as data regarding acceleration, vehicle speed and brakingstatus, lidar and video input, location data and control system statussuch as a program counter indicating operational state of the softwareexecuting during the event. In some embodiments, after an event isdetected by control system 731, downloading of volatile data tonon-volatile data is performed within a specific time duration (e.g.less than five to ten seconds).

In one embodiment, the control system 731 of vehicle 711 may beconfigured to extract event data from the non-volatile memory andanalyze that data. The event data is extracted after being downloadedfollowing detection of an event. For example, the event data can betransmitted wirelessly to a central server that monitors the travel andoperation of vehicle 711.

In one embodiment, only a predetermined portion of system memory 729 isdownloaded. Processor 733 can determine after detection of an event thepredetermined portion to be downloaded. In another embodiment, data fromsystem memory 729 can be downloaded portion by portion in an order basedon priority of data. The priority may be determined by a type of eventdetected by control system 731.

FIG. 6 illustrates a server 101 that stores event data received from avehicle, according to one embodiment. In one example, the event datacorresponds to a determination by a central processing device thatobject data from sensor modules of vehicle 111 do not match. Forexample, vehicle 111 may detect object 155 using sensor modules thatsend object data to the central processing device.

In one example, if a sensing system failure for vehicle 111 is detected,event data 160 (e.g., stored in volatile memory 727 of FIG. 5) isdownloaded from system memory 729 to non-volatile memory 735. Event data160 can include data such as sensor data 103 (obtained from sensormodules of vehicle 111), location data 163, data related to activationof an emergency braking system, or data output from an artificial neuralnetwork. During operation, vehicle 111 collects data regarding objectsthat are detected such as object 155 and object 157.

The event data 160 is extracted from the non-volatile memory andcommunicated through a wireless communication network 102 to a server101. Server 101 analyzes the event data 160 to determine a cause ofsensing failure (e.g., software failure). For example, server 101 canuse emulation to determine whether software executing on the vehicle 111functioned properly. Server 101 can select the type of event data 160(e.g. sensor data or control system status) to be downloaded foranalysis. One example of wireless communication network 102 is a cellphone network.

The collected event data received from the vehicle 111 by server 101 isanalyzed. For example, this analysis can include pattern recognition orother data analysis (e.g., determining a correlation of event data toother data).

In response to identifying a failure of software, at least one action isperformed. For example, server 101 can transmit a communication tovehicle 111 that causes the vehicle to re-configure software.

In one example, a vehicle may communicate, via a wireless connection 115to an access point (or base station) 105, with the server 101 to submitevent data. The wireless connection 115 may be made via a wireless localarea network, a cellular communications network, and/or a communicationlink 107 to a satellite 109 or a communication balloon.

Optionally, the vehicle 111 has a self-learning capability. After anextended period on the road, the vehicle 111 may have its softwarere-configured. In one example, the centralized server 101 may beoperated by a factory, a producer or maker of the vehicle 111 or avendor of the autonomous driving and/or advanced driver assistancesystem for vehicle 111.

FIG. 7 illustrates an example of vehicle 111 including sensor modules137 and configured using an Artificial Neural Network (ANN) model,according to one embodiment. Sensor modules 137 provide object data tocomputer 131. Computer 131 includes a processor and/or a softwareprocess that performs the role of a central processing device.

The vehicle 111 includes an infotainment system 149, a communicationdevice 139, one or more sensors 137, and a computer 131 that isconnected to some controls of the vehicle 111, such as a steeringcontrol 141 for the direction of the vehicle 111, a braking control 143for stopping of the vehicle 111, an acceleration control 145 for thespeed of the vehicle 111, etc.

The computer 131 of the vehicle 111 includes one or more processors 133,memory 135 storing firmware (or software) 127, the ANN model 119, andother data 129.

In one example, firmware 127 is updated by an over-the-air update inresponse to a communication from server 101 sent in response toidentifying faulty software. Alternatively, and/or additionally, otherfirmware of various computing devices or systems of vehicle 111 can beupdated.

The one or more sensors 137 may include a visible light camera, aninfrared camera, a lidar, radar, or sonar system, and/or peripheralsensors, which are configured to provide sensor input to the computer131. A module of the firmware (or software) 127 executed in theprocessor(s) 133 applies the sensor input to an ANN defined by the model119 to generate an output that identifies or classifies an event orobject captured in the sensor input, such as an image or video clip.Data from this identification and/or classification can be included indata collected by a memory device (e.g., memory device 180) and sentfrom a vehicle to server 101 as discussed above.

Alternatively, and/or additionally, the identification of unsafesoftware can be used by an autonomous driving module of the firmware (orsoftware) 127 to generate a response. The response may be a command toactivate and/or adjust one of the vehicle controls 141, 143, and 145. Inone embodiment, the response is an action performed by the vehicle wherethe action has been configured based on an update command from server101 (e.g., the update command can be generated by server 101 in responseto determining that software of vehicle 111 is faulty based on analysisof event data). In one embodiment, prior to generating the controlresponse, the vehicle is configured. In one embodiment, theconfiguration of the vehicle is performed by updating firmware ofvehicle 111. In one embodiment, the configuration of the vehicleincludes updating of the computer model stored in vehicle 111 (e.g., ANNmodel 119).

In one embodiment, the server 101 stores received sensor input as partof sensor data for the subsequent further training or updating of theANN model 119 using the supervised training module 117. When an updatedversion of the ANN model 119 is available in the server 101, the vehicle111 may use the communication device 139 to download the updated ANNmodel 119 for installation in the memory 135 and/or for the replacementof the previously installed ANN model 119. These actions may beperformed in response to determining that vehicle 111 is failing toproperly detect objects and/or in response to identifying unsafesoftware.

In one example, the outputs of the ANN model 119 can be used to control(e.g., 141, 143, 145) the acceleration of a vehicle (e.g., 111), thespeed of the vehicle 111, and/or the direction of the vehicle 111,during autonomous driving.

In one example, data obtained from a sensor of a vehicle may be an imagethat captures an object using a camera that images using lights visibleto human eyes, or a camera that images using infrared lights, or asonar, radar, or lidar system. In one embodiment, image data obtainedfrom at least one sensor of the vehicle is part of the collected datafrom the vehicle that is analyzed. In some instances, the ANN model isconfigured for a particular vehicle based on the sensor and othercollected data.

FIG. 8 shows a method to determine whether object data from a firstsensor module corresponds to object data from a second sensor module,according to one embodiment. For example, the first sensor module issensor module 612 and the second sensor module is sensor module 614. Forexample, the method of FIG. 8 can be implemented in the system of FIGS.1-7.

The method of FIG. 8 can be performed by processing logic that caninclude hardware (e.g., processing device, circuitry, dedicated logic,programmable logic, microcode, hardware of a device, integrated circuit,etc.), software (e.g., instructions run or executed on a processingdevice), or a combination thereof. In some embodiments, the method ofFIG. 8 is performed at least in part by central processing device 618.

Although shown in a particular sequence or order, unless otherwisespecified, the order of the processes can be modified. Thus, theillustrated embodiments should be understood only as examples, and theillustrated processes can be performed in a different order, and someprocesses can be performed in parallel. Additionally, one or moreprocesses can be omitted in various embodiments. Thus, not all processesare required in every embodiment. Other process flows are possible.

In block 601, first object data is received from a first sensor moduleof an autonomous vehicle. For example, object data regarding a positionand/or a type of object is received by central processing device 618from sensor module 612.

In block 603, second object data is received from a second sensor moduleof the autonomous vehicle. For example, object data regarding a positionand/or a type of object is received by central processing device 618from sensor module 614.

In block 605, a comparison is made of the first object data to thesecond object data. For example, central processing device 618 comparesobject data from sensor modules 612, 614 based on a statisticalcorrelation.

In block 607, based on this comparison of the object data, adetermination is made whether the first object data corresponds to thesecond object data. For example, central processing device 618 comparesthe statistical correlation to a predetermined threshold or limit todetermine whether object data from sensor module 612 corresponds to(e.g., matches) object data from sensor module 614.

In block 609, in response to determining that the first object data doesnot correspond to the second object data, an action is performedassociated with operation of the autonomous vehicle. For example,central processing device 618 sends a signal to host processing device620 to cause a change in operation of one or more vehicle systems 622.

In one embodiment, a method for an autonomous vehicle (e.g., autonomousvehicle 610) includes: receiving first object data from a first sensormodule (e.g., sensor module 612); receiving second object data from asecond sensor module (e.g., sensor module 614); comparing the firstobject data to the second object data; determining, based on comparingthe first object data to the second object data, whether the firstobject data corresponds to the second object data; and in response todetermining that the first object data does not correspond to the secondobject data, performing an action for the autonomous vehicle.

In one embodiment, the first sensor module comprises a processing deviceconfigured to process raw data from a sensor.

In one embodiment, the first object data comprises a position and objecttype for an object detected by the first sensor module, and the secondobject data comprises a position and object type for an object detectedby the second sensor module.

In one embodiment, comparing the first object data to the second objectdata comprises determining a correlation of the first object data to thesecond object data.

In one embodiment, a non-transitory computer storage medium storesinstructions which, when executed on a computing device (e.g., centralprocessing device 618), cause the computing device to perform a methodfor an autonomous vehicle, the method comprising: receiving first objectdata from a first sensor module; receiving second object data from asecond sensor module; comparing the first object data to the secondobject data; determining, based on comparing the first object data tothe second object data, whether the first object data corresponds to thesecond object data; and in response to determining that the first objectdata does not correspond to the second object data, performing an actionassociated with the autonomous vehicle.

In one embodiment, a processing device controls navigation of theautonomous vehicle, and performing the action comprises causing theprocessing device to change a path of navigation for the autonomousvehicle.

In one embodiment, an autonomous vehicle comprises: a first sensormodule comprising a first sensor (e.g., sensor 624) and a firstprocessing device (e.g., processor 626), wherein the first processingdevice evaluates sensor data (e.g., raw data in FIG. 2) from the firstsensor to provide first object data; a second sensor module comprising asecond sensor and a second processing device, wherein the secondprocessing device evaluates sensor data from the second sensor toprovide second object data; a central processing device configured toreceive object data provided by sensor modules of the autonomousvehicle; and memory storing instructions configured to instruct thecentral processing device to: receive the first object data; receive thesecond object data; make a comparison of the first object data to thesecond object data; determine, based on the comparison, whether thefirst object data corresponds to the second object data; and in responseto determining that the first object data does not correspond to thesecond object data, perform an action associated with operation of theautonomous vehicle.

In one embodiment, the first object data includes a type of objectselected from a plurality of object types.

In one embodiment, the first sensor module further includes first memory(e.g., memory 628) storing data for a neural network (e.g., neuralnetwork 632), and the first processing device uses the neural network toevaluate the sensor data from the first sensor.

In one embodiment, making the comparison of the first object data to thesecond object data comprises performing a statistical correlation of thefirst object data to the second object data.

In one embodiment, the instructions are further configured to instructthe central processing device to determine a context of the autonomousvehicle based on sensor data from at least one sensor other than thefirst sensor and the second sensor, and determining whether the firstobject data corresponds to the second object data is based in part onthe context.

In one embodiment, the autonomous vehicle further comprises a hostprocessing device, performing the action comprises sending a signal tothe host processing device, and sending the signal causes the hostprocessing device to change a configuration of a vehicle system of theautonomous vehicle.

In one embodiment, the first sensor is a lidar or radar sensor, and thesecond sensor is a camera.

In one embodiment, the first processing device is a system on chip, afield-programmable gate array, a graphics processing unit, or anapplication specific integrated circuit.

In one embodiment, the first object data comprises a position of anobject detected by the first sensor module, and the second object datacomprises a position of an object detected by the second sensor module.

In one embodiment, determining whether the first object data correspondsto the second object data comprises comparing a number of times that thefirst object data matches the second object data to a threshold.

In one embodiment, the instructions are further configured to instructthe central processing device to determine whether a current time forthe autonomous vehicle corresponds to daytime or nighttime, and thethreshold is a first threshold during the daytime, and is a secondthreshold during the nighttime.

In one embodiment, the threshold is a predetermined percentage of atotal number of comparisons.

In one embodiment, performing the action associated with operation ofthe autonomous vehicle comprises sending a signal that causes a hostprocessing device of the autonomous vehicle to perform a diagnostic testof at least one of the first sensor module or the second sensor module.

In one embodiment, the first object data includes a first type ofdetected object, wherein the second object data includes a second typeof detected object, and making the comparison of the first object datato the second object data comprises comparing the first type to thesecond type.

In one example, the central or host processing device can be one or moregeneral-purpose processing devices such as a microprocessor, a centralprocessing unit, or the like. More particularly, the central or hostprocessing device can be a complex instruction set computing (CISC)microprocessor, reduced instruction set computing (RISC) microprocessor,very long instruction word (VLIW) microprocessor, or a processorimplementing other instruction sets, or processors implementing acombination of instruction sets. The central or host processing devicecan also be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. The central or host processing device is configured toexecute instructions for performing the operations and steps discussedherein.

FIG. 9 shows an autonomous vehicle 303 controlled and/or configured inresponse to determining that object data from a first sensor module ofthe vehicle does not correspond to object data from a second sensormodule of the vehicle, according to one embodiment. Sensor modules 306provide object data for comparison to central processing device 618.Sensor modules 306 are an example of sensor modules 612, 614.

Based on comparing the object data from sensor modules 306, Centralprocessing device 618 sends a signal to a controller 307. Controller 307is an example of host processing device 620. In one example, in responseto receiving the signal, controller 307 causes a display to a user onone or more display devices 308. In one example, the display presentsinformation in a user interface indicating to the user that a failurehas been detected. In one example, the presentation indicates thecomponents identified as having failed. Display device 308 is an exampleof a vehicle system 622.

Memory 309 stores a computer model 312 and training data 314. Computermodel 312 is an example of neural network 632. In one example, computermodel 312 and/or training data 314 are used to configure and/or updateone or more of sensor modules 306. In one example, an update forcomputer model 312 and/or training data 314 is received from server 301.

Systems of vehicle 303 include display device 308 or other device,system, or component. In one example, controller 307 is included in amemory device 180, and controller 307 controls collection of data fromvarious systems of the vehicle 303.

Server 301 may store, for example, event data 160 collected bycontroller 307 and sent to server 301. The event data may include objectdata provided by Central processing device 618.

Server 301 may determine, using event data 160 and/or other dataprovided by central processing device 618, that a sensor componentand/or sensor software (e.g., executing on central processing device618) of vehicle 303 is operating improperly. In response to thisdetermination, server 301 may cause the controller 307 to terminate anautonomous navigation mode. Other actions can be performed in responseto this determination including, for example, configuring vehicle 303 byupdating firmware 304, updating computer model 312, updating data indatabase 310, and/or updating training data 314.

The controller 307 may receive data collected by one or more sensormodules 306. The sensor modules 306 may be, for example, mounted in theautonomous vehicle 303. The sensor modules 306 may include, for example,a camera, a microphone, a motion detector, and/or a camera. At least aportion of the sensor modules may provide data associated with objectsnewly detected by vehicle 303 during travel.

The sensor modules 306 may provide various types of data for collectionby the controller 307 and/or central processing device 618. For example,the collected data may include image data from the camera and/or audiodata from the microphone.

In one embodiment, the central processing device 618 and/or controller307 analyzes the collected data from the sensor modules 306. Theanalysis of the collected data includes providing some or all of thecollected data as one or more inputs to a computer model 312. Thecomputer model 312 can be, for example, an artificial neural networktrained by deep learning. In one example, the computer model is amachine learning model that is trained using training data 314. Thecomputer model 312 and/or the training data 314 can be stored, forexample, in memory 309. An output from the computer model 312 can betransmitted to server 301 as part of object data.

In one embodiment, memory 309 stores a database 310, which may includedata collected by sensor modules 306 and/or data received by acommunication interface 305 from computing device, such as, for example,a server 301. In one example, this communication may be used towirelessly transmit collected data from the sensor modules 306 to theserver 301. The received data may include configuration, training, andother data used to configure control of the display devices 308 or othervehicle systems by controller 307. The received data may also be used toupdate a configuration of a machine learning model stored in memory 309as computer model 312.

In one embodiment, firmware 304 controls, for example, the operations ofthe controller 307 in controlling components of vehicle 303. Thecontroller 307 also can, for example, run the firmware 304 to performoperations responsive to communications from the server 301. Theautonomous vehicle 303 includes volatile Dynamic Random-Access Memory(DRAM) 311 for the storage of run-time data and instructions used by thecontroller 307. The run-time data and/or instructions can be part of thedata downloaded in response to detecting an event.

At least some embodiments of the systems and methods disclosed hereincan be implemented using computer instructions (e.g., firmware) executedby a controller or processor. In some instances, hardware circuits canbe used to implement at least some of the functions of the firmware. Thefirmware can be initially stored in non-volatile storage media oranother non-volatile device, and loaded into volatile DRAM and/or thein-processor cache memory for execution. In one example, the firmwarecan be configured to control collection of data from a vehicle.

FIG. 10 is a block diagram of an autonomous vehicle (e.g., vehicle 610)including one or more various components and/or subsystems, each ofwhich can be updated in various embodiments to configure the vehicleand/or perform other actions associated with the vehicle (e.g., changingoperation of a vehicle system, updating a configuration, and/or otheractions performed in response to identifying a sensor-related failurebased on comparison of object data provided by sensor modules). Thesystem illustrated in FIG. 10 may be installed entirely within avehicle.

The system includes an autonomous vehicle subsystem 402. Subsystem 402is an example of vehicle system 622 of FIG. 1. In the illustratedembodiment, autonomous vehicle subsystem 402 includes map database 402A,radar devices 402B, lidar devices 402C, digital cameras 402D, sonardevices 402E, GPS receivers 402F, and inertial measurement units 402G.Each of the components of autonomous vehicle subsystem 402 comprisestandard components provided in most current autonomous vehicles. In oneembodiment, map database 402A stores a plurality of high-definitionthree-dimensional maps used for routing and navigation. Radar devices402B, lidar devices 402C, digital cameras 402D, sonar devices 402E, GPSreceivers 402F, and inertial measurement units 402G may comprise variousrespective devices installed at various positions throughout theautonomous vehicle as known in the art. For example, these devices maybe installed along the perimeter of an autonomous vehicle to providelocation awareness, collision avoidance, and other standard autonomousvehicle functionality.

Vehicular subsystem 406 is additionally included within the system.Vehicular subsystem 406 includes various anti-lock braking systems 406A,engine control units 402B, and transmission control units 402C. Thesecomponents may be utilized to control the operation of the autonomousvehicle in response to the streaming data generated by autonomousvehicle subsystem 402. The standard autonomous vehicle interactionsbetween autonomous vehicle subsystem 402 and vehicular subsystem 406 aregenerally known in the art and are not described in detail herein.

The processing side of the system includes one or more processors 410,short-term memory 412, an RF subsystem 414, graphics processing units(GPUs) 416, long-term storage 418 and one or more interfaces 420.

The one or more processors 410 may comprise central processing units,FPGAs, or any range of processing devices needed to support theoperations of the autonomous vehicle. Memory 412 comprises DRAM or othersuitable volatile RAM for temporary storage of data required byprocessors 410. RF system 414 may comprise a cellular transceiver and/orsatellite transceiver. Long-term storage 418 may comprise one or morehigh-capacity solid-state drives (SSDs). In general, long-term storage418 may be utilized to store, for example, high-definition maps, routingdata, and any other data requiring permanent or semi-permanent storage.GPUs 416 may comprise one more high throughput GPU devices forprocessing data received from autonomous vehicle subsystem 402. Finally,interfaces 420 may comprise various display units positioned within theautonomous vehicle (e.g., an in-dash screen).

The system additionally includes a reporting subsystem 404 whichperforms data collection (e.g., collection of data obtained from sensorsof the vehicle that is used to drive the vehicle). The reportingsubsystem 404 includes a sensor monitor 404A which is connected to bus408 and records sensor data transmitted on the bus 408 as well as anylog data transmitted on the bus. The reporting subsystem 404 mayadditionally include one or more endpoints to allow for systemcomponents to transmit log data directly to the reporting subsystem 404.

The reporting subsystem 404 additionally includes a packager 404B. Inone embodiment, packager 404B retrieves the data from the sensor monitor404A or endpoints and packages the raw data for transmission to acentral system (illustrated in FIG. 11). In some embodiments, packager404B may be configured to package data at periodic time intervals.Alternatively, or in conjunction with the foregoing, packager 404B maytransmit data in real-time and may compress data to facilitate real-timecommunications with a central system.

The reporting subsystem 404 additionally includes a batch processor404C. In one embodiment, the batch processor 404C is configured toperform any preprocessing on recorded data prior to transmittal. Forexample, batch processor 404C may perform compression operations on thedata prior to packaging by packager 404B. In another embodiment, batchprocessor 404C may be configured to filter the recorded data to removeextraneous data prior to packaging or transmittal. In anotherembodiment, batch processor 404C may be configured to perform datacleaning on the recorded data to conform the raw data to a formatsuitable for further processing by the central system.

Each of the devices is connected via a bus 408. Bus 408 is an example ofbus 174 of FIG. 3. In one embodiment, the bus 408 may comprise acontroller area network (CAN) bus. In some embodiments, other bus typesmay be used (e.g., a FlexRay or MOST bus). Additionally, each subsystemmay include one or more additional busses to handle internal subsystemcommunications (e.g., LIN busses for lower bandwidth communications).

FIG. 11 is a block diagram of a centralized autonomous vehicleoperations system, according to various embodiments. As illustrated, thesystem includes a number of autonomous vehicles 502A-502E. Each vehicleis an example of vehicle 610.

In one embodiment, each autonomous vehicle may comprise an autonomousvehicle such as that depicted in FIG. 10. Each autonomous vehicle502A-502E may communicate with a central system 514 via a network 516.In one embodiment, network 516 comprises a global network such as theInternet. In one example, central system 514 is implemented using one ormore of servers 101 or 301.

The system additionally includes a plurality of client devices 508A,508B. In the illustrated embodiment, client devices 508A, 508B maycomprise any personal computing device (e.g., a laptop, tablet, mobilephone, etc.). Client devices 508A, 508B may issue requests for data fromcentral system 514. In one embodiment, client devices 508A, 508Btransmit requests for data to support mobile applications or web pagedata, as described previously.

In one embodiment, central system 514 includes a plurality of servers504A. In one embodiment, servers 504A comprise a plurality of front endwebservers configured to serve responses to client device 508A, 508B.The servers 504A may additionally include one or more applicationservers configured to perform various operations to support one or morevehicles.

In one embodiment, central system 514 additionally includes a pluralityof models 504B. In one embodiment, models 504B may store one or moreneural networks for classifying autonomous vehicle objects. The models504B may additionally include models for predicting future events. Insome embodiments the models 504B may store a combination of neuralnetworks and other machine learning models.

Central system 514 additionally includes one or more databases 504C. Thedatabases 504C may include database record for vehicles 504D,personalities 504E, and raw data 504F. Raw data 504F may comprise anunstructured database for storing raw data received from sensors andlogs as discussed previously.

The present disclosure includes methods and apparatuses which performthe methods described above, including data processing systems whichperform these methods, and computer readable media containinginstructions which when executed on data processing systems cause thesystems to perform these methods.

A server and/or a computing device of a vehicle above can be implementedas one or more data processing systems. A typical data processing systemmay include an inter-connect (e.g., bus and system core logic), whichinterconnects a microprocessor(s) and memory. The microprocessor istypically coupled to cache memory.

The inter-connect interconnects the microprocessor(s) and the memorytogether and also interconnects them to input/output (I/O) device(s) viaI/O controller(s). I/O devices may include a display device and/orperipheral devices, such as mice, keyboards, modems, network interfaces,printers, scanners, video cameras and other devices known in the art. Inone embodiment, when the data processing system is a server system, someof the I/O devices, such as printers, scanners, mice, and/or keyboards,are optional.

The inter-connect can include one or more buses connected to one anotherthrough various bridges, controllers and/or adapters. In one embodimentthe I/O controllers include a USB (Universal Serial Bus) adapter forcontrolling USB peripherals, and/or an IEEE-1394 bus adapter forcontrolling IEEE-1394 peripherals.

The memory may include one or more of: ROM (Read Only Memory), volatileRAM (Random Access Memory), and non-volatile memory, such as hard drive,flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In the present disclosure, some functions and operations are describedas being performed by or caused by software code to simplifydescription. However, such expressions are also used to specify that thefunctions result from execution of the code/instructions by a processor,such as a microprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited tonon-transitory, recordable and non-recordable type media such asvolatile and non-volatile memory devices, read only memory (ROM), randomaccess memory (RAM), flash memory devices, floppy and other removabledisks, magnetic disk storage media, optical storage media (e.g., CompactDisk Read-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.),among others. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

The above description and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described in order to avoidobscuring the description. References to one or an embodiment in thepresent disclosure are not necessarily references to the sameembodiment; and, such references mean at least one.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. A system comprising: a central processing deviceconfigured to receive object data provided by sensors of a vehicle; andmemory storing instructions configured to instruct the centralprocessing device to: receive first object data based on a first objectdetection by a first sensor of the vehicle; receive second object databased on a second object detection by a second sensor of the vehicle;make a comparison of the first object data to the second object data,the comparison comprising performing a correlation of the first objectdata to the second object data; determine, based on the comparison ofthe first object data to the second object data, whether the firstobject data corresponds to the second object data; and in response todetermining that the first object data does not correspond to the secondobject data, change a configuration of at least one of the first sensoror the second sensor.
 2. The system of claim 1, wherein the instructionsare further configured to instruct the central processing device to: inresponse to determining that the first object data does not correspondto the second object data, perform a diagnostic test of at least one ofthe first sensor or the second sensor; wherein the configuration of thefirst sensor or the second sensor is changed based on the diagnostictest.
 3. The system of claim 2, wherein the instructions are furtherconfigured to instruct the central processing device to determine, basedon the diagnostic test, that a malfunction exists in at least one of thefirst sensor or the second sensor.
 4. The system of claim 1, wherein theinstructions are further configured to instruct the central processingdevice to, in response to determining that the first object data doesnot correspond to the second object data, disable an autonomous mode ofoperation for the vehicle.
 5. The system of claim 1, wherein thecorrelation of the first object data to the second object data providesa result, and the instructions are further configured to instruct thecentral processing device to: determine an expected correlation; andcompare the result to the expected correlation.
 6. The system of claim5, wherein: the instructions are further configured to instruct thecentral processing device to determine, based on at least one of thefirst object data or the second object data, a type of object; and theexpected correlation is determined based on the type of object.
 7. Thesystem of claim 1, wherein determining whether the first object datacorresponds to the second object data comprises comparing a number oftimes that the first object data matches the second object data to athreshold.
 8. The system of claim 7, wherein the threshold varies basedon a time of day.
 9. A vehicle comprising: a first sensor configured tostore a first object type; a second sensor configured to store a secondobject type; and a central processing device configured to: receivefirst object data from the first sensor; receive second object data fromthe second sensor; make a comparison, based on the first object type orthe second object type, of the first object data to the second objectdata; determine, based on the comparison of the first object data to thesecond object data, whether the first object data corresponds to thesecond object data; and in response to determining that the first objectdata does not correspond to the second object data, change aconfiguration of at least one of the first sensor or the second sensor.10. The vehicle of claim 9, wherein making the comparison of the firstobject data to the second object data comprises comparing the firstobject type to the second object type.
 11. The vehicle of claim 9,wherein the comparison of the first object data to the second objectdata comprises performing a correlation of the first object data to thesecond object data.
 12. The vehicle of claim 11, wherein the correlationis based on a type of object.
 13. The vehicle of claim 9, wherein thefirst object data comprises a position of an object detected by thefirst sensor, and the second object data comprises a position of anobject detected by the second sensor.
 14. The vehicle of claim 9,wherein the central processing device is further configured to:determine a context of the vehicle based on data from at least onesensor other than the first sensor and the second sensor; whereindetermining whether the first object data corresponds to the secondobject data is based in part on the context.
 15. The vehicle of claim 9,wherein the first sensor includes memory storing data for a neuralnetwork, and wherein the first sensor uses the neural network toevaluate data collected by the first sensor.
 16. The vehicle of claim 9,wherein: the determining comprises comparing a number of times that thefirst object data matches the second object data to a threshold; and thethreshold is a predetermined percentage of a total number ofcomparisons.
 17. A vehicle comprising: a lidar sensor; a camera; and acentral processing device configured to: receive first object data fromthe lidar sensor; receive second object data from the camera; make acomparison of the first object data to the second object data;determine, based on the comparison of the first object data to thesecond object data, whether the first object data corresponds to thesecond object data; and in response to determining that the first objectdata does not correspond to the second object data, change aconfiguration of at least one of the lidar sensor or the camera.
 18. Thevehicle of claim 17, wherein determining whether the first object datacorresponds to the second object data comprises determining whether astatistical detection relationship is maintained between the firstobject data and the second object data.
 19. The vehicle of claim 17,wherein the central processing device is further configured to, inresponse to determining that the first object data does not correspondto the second object data, require a person to take over control of thevehicle.
 20. The vehicle of claim 17, wherein the central processingdevice is further configured to: perform a correlation of the firstobject data to the second object data to provide a result; determine anexpected correlation; and compare the result to the expectedcorrelation.
 21. The vehicle of claim 20, wherein: the centralprocessing device is further configured to determine, based on at leastone of the first object data or the second object data, a type ofobject; and the expected correlation is determined based on thedetermined type of object.